Computer source code is typically written using a high level computer language. Examples of high level languages include C, C++, the Java™ programming language, etc. Once written, source code is converted into object or machine code, which can be executed by a computer. A computer program called a compiler program receives as input source code and generates as output object code which may be loaded into the computer memory and executed.
Often, the source code will contain “bugs”. Bugs are errors in the original source code that cause unexpected results during the execution of the object code. These unexpected results may cause the computer executing the object code to generate unintended output and erroneous data, or to terminate prematurely.
Before releasing any set of computer code, software manufacturers test their code rigorously. Ideally, every set of code should be 100% tested prior to release so that once released, the computer code will not experience any errors. From a practical standpoint, however, this is extremely difficult if not impossible. To do so would require more resources (both in terms of human resources and actual costs) than would be practicable to expend.
One of the most difficult aspects of testing software is to determine when a set of code has been sufficiently tested. To determine whether a set of code has been sufficiently tested, manufacturers generally run tests on the code, and generate test coverage data based upon the tests. Basically, the coverage data specifies which lines of source code were executed during the tests, and what percentage of the lines of source code (LOC) were actually executed and tested. Based upon the coverage data, the manufacturer determines whether the set of code has been sufficiently tested.
However, merely determining that a certain percentage of LOC executed during testing does not reveal the actual quality of testing. For example, it may be that the LOC that did not execute during testing were not that critical. However, even if the percentage of LOCs that executed is high, some very important LOCs may still have been missed. Moreover, the executed percentage does not reveal what further tests should be developed to cover the remaining LOCs.
It is possible to perform a detailed analysis of the LOCs that executed during testing to determine if the test coverage is adequate and/or what additional tests should be performed. However, frequently the test engineers who conduct the testing do not have sufficient understanding of the code to perform such analysis. The software engineer(s) who wrote the software under test are better able to analyze the test results. However, this analysis takes considerable time and often the software engineers do not have the time to analyze the test results.
Thus, it is difficult to properly analyze test results, such as which LOCs executed. As a consequence, resources are frequently wasted because of the difficulty in determining how to design further tests after an initial test run. Moreover, resources are sometimes wasted in testing code, or at least portions thereof, which do not require further testing. As software becomes increasingly complex these problems become more difficult.
Because of these and potentially other drawbacks, this approach does not provide wholly satisfactory results. Consequently, an improved method and system for testing computer software is desired.